During ISO 26262 development, test is a critical component. Safety-critical systems must react properly to test scenarios and stay within specified safety limits when exposed to various human and environmental inputs. Using high quality test systems can improve a product’s performance, increase quality and reliability, and lower return rates. It is estimated that the cost of a failure decreases by 10 times when the error is caught in production instead of in the field and decreases 10 times again if it is caught in design instead of production. By catching these defects and collecting the data to improve a design or process, test delivers value to your organization. Driving innovation into this process through technology insertion and best-practice methodologies can generate large efficiency gains and cost reductions. It is easy to look past the tools and think only about the design of the system, but in reality the tools are very important to the safety of the end user.

ISO 26262 recognizes that using widely accepted software tools simplifies or automates activities and tasks required for the development of electrical, electronic, and software elements that provide safety-related functions. Before explaining the details of the tool qualification process, it is important to define an important part of tool qualification, the Tool Confidence Level.

From the inputs and outputs of the tool, typical (or reference) use cases are developed. The analysis of these use cases leads to the determination of the Tool Confidence Level, or TCL. The TCL and ASIL determine the level of qualification required for the software tool. Two specific areas are evaluated to determine the confidence level:

The Tool Confidence Level is determined to be TCL1, TCL2, TCL3, or TCL4, with TCL4 being the highest level of confidence and TCL1 being the lowest level of confidence.

In order to qualify a tool under ISO 26262, there are many requirements. For instance, the ASIL must already be determined. The tool must have a user manual,  a unique identification and version number, a description of the features, installation process, and environment (to name a few). ISO 26262 requires the following tool qualification work products:

The Software tool Qualification Plan (STQP) is created early in the development life cycle of the safety-related item. It focuses on two areas: planning for the qualification of a software tool, and listing the use-cases that demonstrate the tool is classified with the required level of confidence.

The STQP must include items such as a unique identification and version number of the software tool, use cases, the environment, description, user manual, and the pre-defined ASIL.

The main purpose of the Software Tool Classification Analysis (STCA) is to determine the Tool Confidence Level. There are two main components that determine the TCL. The first is the Tool Impact (TI). The second is the Tool Error Detection (TD). Based on these two components, the appropriate TCL is chosen.

TI1 or TI2 are the two classes of Tool Impact. TI1 is chosen when there is an argument that there is no possibility that the malfunctioning software tool can violate a safety requirement. For all other cases, TI2 is chosen.

For example, let us say that a tool produces a typo in the documentation for a particular software function. This can be considered a nuisance only, and does not violate the safety requirement under test. This would results in a tool impact of TI1. If the tool produces an error that could change the behavior of the system in any way, then TI2 will be chosen.

The Tool Error Detection is classified as TD1 through TD3. TD1 is chosen if there is a high degree of confidence in the tool's ability to detect an error where TD3 is chosen for a very low degree of confidence, often when it is determined that the error can only be detected randomly.

For example, a software tool might check a design model for errors. In this case, static analysis of the model is performed. While static analysis is good, it cannot check all possible violations in the model. It is also important to note that this does not necessarily imply that the model is incorrect; it simply means that additional testing is needed. This scenario results in a ‘medium’ degree of confidence, or TD2.

Tool Error Detection

TD1

TD2

TD3

Tool Impact

TI1

TCL1

TCL1

TCL1

TI2

TCL1

TCL2

TCL3

Once the Tool Impact (TI) and Tool Error Detection (TD) are determined, a value of TCL 1 to TCL 3 is given, depending on required level of confidence. Sometimes multiple use cases can result in multiple TCLs. In this case, the highest TCL is used. For each software tool, the user needs to carry out the tool classification.  

Several pieces of information must be provided to ensure proper usage of the software tool.

The Software Tool Qualification Report contains the results and evidence that the tool qualification was completed and requirements fulfilled. Any malfunctions or erroneous outputs during validation should be analyzed and documented here.

An important aspect of tool qualification is the concept of increased confidence from use. If the qualification requirements can already be demonstrated for a given tool, then further qualification is no longer needed. This can dramatically save cost and time throughout the development process. However, qualification requirements must be demonstrated for each safety-related item or element before used in development of that item. In order to demonstrate this, the tool must demonstrate that:

For example, let us say that test tool A was used for validating requirements for car X’s ECU (Engine Control Unit). If test tool A has not violated any safety requirements and remains unchanged, then it can be used to validate car Y’s ECU given that car Y’s ECU is being used in similar manner as car X's ECU.